NPS-AM-1 0-050 


EXCERPT  FROM  THE 

PROCEEDINGS 


OF  THE 


SEVENTH  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 
THURSDAY  SESSIONS 
VOLUME  II 


Acquisition  Research 
Creating  Synergy  for  I  nformed  Change 

May  12  -  13,  2010 

Published:  30  April  2010 


Approved  for  public  release,  distribution  unlimited. 

Prepared  for:  Naval  Postgraduate  School,  Monterey,  California  93943 


llVjpo]  I  ACQUISITION  RESEARCH  PROGRAM 

graduate  school  of  business  &  PUBLIC  policy 

NAVAL  POSTGRADUATE  SCHOOL 


1 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

MAY  2010 


2.  REPORT  TYPE 


4.  TITLE  AND  SUBTITLE 

When  More  is  Better:  Design  Principles  for  Prediction  Markets  in 
Defense  Acquisition  Cost  Forecasting 


6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS (ES) 

Massachusetts  Institute  of  Technology, Lean  Advancement  Initiative, 77 
Massachusetts  Avenue, Cambridge, MA, 02139 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS  (ES) 


3.  DATES  COVERED 

00-00-2010  to  00-00-2010 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

7th  Annual  Acquisition  Research  Symposium  to  be  held  May  12-13,  2010  in  Monterey,  California.  U.S. 
Government  or  Federal  Rights  License 

14.  ABSTRACT 

This  paper  discusses  the  applicability  of  prediction  markets  in  Defense  Acquisition  projects,  specifically  in 
estimating  their  cost  and  schedule.  Several  temporal  and  political  factors  can  sometimes  limit  the 
effectiveness  of  traditional  methods  of  project  tracking  and  cost  estimation,  which  may  be  overcome  by 
using  prediction  markets.  A  prediction  market  provides  an  environment  for  traders  to  buy  and  sell 
contracts  whose  values  are  tied  to  uncertain  future  events.  Efficient  prediction  markets  have  been  shown  to 
outperform  available  polls  and  other  forecasting  mechanisms.  There  are  various  prediction  markets  based 
on  different  models  and  algorithms.  Our  focus  is  not  to  analyze  these  models,  but  to  identify  the  design 
principles  of  implementing  a  proven  prediction  market  model  in  a  defense  acquisition  project.  Some  pilot 
studies  have  been  carried  out  that  provide  insight  into  the  behavior  of  the  market  participants.  We  found 
increased  involvement  of  participants  and  greater  interest  in  the  projects  to  be  major  benefits.  The  areas 
that  need  to  be  considered  in  the  design  and  implementation  of  markets  are  related  to  the  participants 
(like,  which  traders  to  include);  the  information  to  be  collected  or  the  stocks;  the  marketplace  to  be  used; 
and  the  incentive  structure  to  keep  the  participants  motivated  to  trade. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

58 

Report  (SAR) 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


The  research  presented  at  the  symposium  was  supported  by  the  Acquisition  Chair  of  the 
Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School. 


To  request  Defense  Acquisition  Research  or  to  become  a  research  sponsor,  please 
contact: 

NPS  Acquisition  Research  Program 
Attn:  James  B.  Greene,  RADM,  USN,  (Ret.) 

Acquisition  Chair 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

555  Dyer  Road,  Room  332 

Monterey,  CA  93943-5103 

Tel:  (831)656-2092 

Fax:  (831)656-2253 

E-mail:  ibqreene@nps.edu 

Copies  of  the  Acquisition  Sponsored  Research  Reports  may  be  printed  from  our  website 
www.acquisitionresearch.net 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


2 


When  More  is  Better:  Design  Principles  for 
Prediction  Markets  in  Defense  Acquisition  Cost 
Forecasting 


Taroon  Aggarwal — Taroon  Aggarwal  is  a  graduate  student  at  MIT,  doing  his  Masters  in  Systems 
Design  and  Management.  He  is  also  working  as  a  research  assistant  at  MIT  Lean  Advancement 
Initiative  (LAI).  His  research  interests  lie  in  the  areas  of  cost  estimation,  software  engineering  and 
application  of  collective  intelligence  tools  to  large  organizations.  Prior  to  his  graduate  studies,  he 
worked  for  around  five  years  as  an  IT  consultant  in  the  financial  services  domain.  Taroon  also  holds  a 
Bachelor’s  degree  in  engineering  from  India. 

Taroon  Aggarwal 

Research  Assistant 

Lean  Advancement  Initiative 

Massachusetts  Institute  of  Technology 

tar_agg@mit.edu 

Ricardo  Valerdi — Ricardo  Valerdi,  PhD,  is  a  Research  Associate  at  MIT.  He  received  his  doctoral 
degree  in  systems  engineering  from  USC  in  2005,  where  he  developed  the  COSYSMO  model  for 
systems  engineering  cost  estimation.  He  is  also  a  Senior  Member  of  the  Technical  Staff  at  the 
Aerospace  Corporation.  Prior  to  his  doctoral  studies,  Dr.  Valerdi  was  a  systems  engineer  at  Motorola 
Radio  Network  Solutions  Group.  He  served  on  the  INCOSE  Board  of  Directors  as  Treasurer  and  is 
co-Editor-in-Chief  of  the  Journal  of  Enterprise  Transformation. 

Dr.  Ricardo  Valerdi 
Research  Associate 
Lean  Advancement  Initiative 

Center  for  Technology,  Policy,  and  Industrial  Development 
Massachusetts  Institute  of  Technology 
rvalerdi@mit.edu 

Matthew  Potoski — Matthew  Potoski  is  an  Associate  Professor  in  the  Department  of  Political 
Science  at  Iowa  State  University,  where  he  teaches  courses  on  public  management  and  policy.  He 
has  received  Iowa  State  University  LAS  awards  for  Early  and  Mid-Career  Achievement  in  Research. 
He  is  Co-Editor  of  the  Journal  of  Policy  Analysis  and  Management  and  the  International  Public 
Management  Journal.  Dr.  Potoski’s  research  investigates  public  management  and  policy  in  domestic 
and  international  contexts,  including  public  sector  contracting  and  service  delivery,  environmental 
policy,  and  voluntary  regulations.  He  is  co-author  with  Aseem  Prakash  of  the  Voluntary 
Environmentalists  (Cambridge,  2006)  and  Co-Editor  of  Voluntary  Programs:  A  Club  Theory  Approach 
(MIT).  Dr.  Potoski  received  a  PhD  in  Political  Science  from  Indiana  University  in  1998  and  a 
Bachelor’s  degree  in  Government  from  Franklin  and  Marshall  College. 

Dr.  Matthew  Potoski 
Public  Policy  and  Administration 
Department  of  Political  Science 
Iowa  State  University 
potos  ki@iastate.edu 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


443 


Acknowledgements 

This  research  is  supported  by  the  Acquisition  Research  Program  in  the  Graduate 
School  of  Business  and  Public  Policy,  Naval  Postgraduate  School.  We  also  wish  to  thank 
Dr.  Keith  Snider  for  his  efforts  on  behalf  of  the  Acquisition  Research  Program. 

Special  thanks  to  MITRE  for  extending  their  support  and  time,  and  for  providing 
some  useful  information  on  prediction  markets. 

We  wish  to  thank  Adam  Siegel  of  Inkling  Markets  for  his  suggestions  on  prediction 
market  design. 

Abstract 

This  paper  discusses  the  applicability  of  prediction  markets  in  Defense  Acquisition 
projects,  specifically  in  estimating  their  cost  and  schedule. 

Several  temporal  and  political  factors  can  sometimes  limit  the  effectiveness  of 
traditional  methods  of  project  tracking  and  cost  estimation,  which  may  be  overcome  by  using 
prediction  markets.  A  prediction  market  provides  an  environment  for  traders  to  buy  and  sell 
contracts  whose  values  are  tied  to  uncertain  future  events.  Efficient  prediction  markets  have 
been  shown  to  outperform  available  polls  and  other  forecasting  mechanisms. 

There  are  various  prediction  markets  based  on  different  models  and  algorithms.  Our 
focus  is  not  to  analyze  these  models,  but  to  identify  the  design  principles  of  implementing  a 
proven  prediction  market  model  in  a  defense  acquisition  project.  Some  pilot  studies  have 
been  carried  out  that  provide  insight  into  the  behavior  of  the  market  participants.  We  found 
increased  involvement  of  participants  and  greater  interest  in  the  projects  to  be  major 
benefits.  The  areas  that  need  to  be  considered  in  the  design  and  implementation  of  markets 
are  related  to  the  participants  (like,  which  traders  to  include);  the  information  to  be  collected, 
or  the  stocks;  the  marketplace  to  be  used;  and  the  incentive  structure  to  keep  the 
participants  motivated  to  trade. 

Introduction 

Historically,  defense  programs  for  weapons  acquisition  have  taken  longer,  cost  more 
and  delivered  fewer  quantities  and  capabilities  than  planned  (GAO,  2006). 

Some  of  the  reasons  for  such  discrepancies  include: 

•  Lack  of  discipline  in  estimation.  Sometimes  all  the  required  factors  or 
experienced  people  are  not  consulted  during  the  estimation  process.  There  is 
also  a  lack  of  transparency  and  accountability  (GAO,  2006). 

•  Unrealistic  expectations.  The  estimates  can  sometimes  be  too  optimistic 
(GAO,  2008). 

•  Initial  estimates  tend  to  “anchor"  expectations  (Aranda  &  Easterbrook,  2005). 

Initial  figures  of  estimates  set  too  high  expectations  from  the  start,  which  is 
difficult  to  meet  later  on.  Also,  it  is  later  on  difficult  to  incorporate  any 
modifications. 

A  recent  study  by  RAND  involved  an  analysis  of  data  in  the  Selected  Acquisition 
Reports  (SARs)  for  a  sample  of  68  completed  programs.  The  results  showed  that  the 
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average  cost  growth  (including  both  cost  overruns  and  cost  under  runs)  for  these  programs 
was  46%  over  the  baseline  estimate  made  at  Milestone  B  and  16%  over  the  baseline 
estimate  at  Milestone  C.  While  cost  growth  occurred  earlier  in  acquisition  projects,  the 
development  cost  growth  at  completion  for  programs  initiated  in  1970s,  1980s  and  1990s 
remained  relatively  steady  (Arena,  Leonard,  Murray  &  Younossi,  2007). 

It  is  also  worthwhile  to  note  that  of  the  23  programs  assessed  in  the  GAO  report 
(GAO,  2006),  around  10  are  expecting  development  cost  overruns  greater  than  30%,  or 
have  delayed  the  delivery  of  initial  operational  capability  by  at  least  1  year.  The  impact  of 
such  conditions  is  the  reduction  in  the  value  of  the  DoD’s  defense  dollars  and  lower  return 
on  investment  (ROI)  (GAO,  2006).  See  Table  4  in  the  Appendix  for  more  details. 

The  key  recommendation  for  the  executive  action  from  the  report  is  to  ensure  that 
appropriate  knowledge  is  captured  and  used  at  critical  junctures,  specifically  at  the  following 
key  points: 

•  Program  start, 

•  Design  review  for  transitioning  from  system  integration  to  system 

demonstration,  and 

•  Production  commitment. 

An  effective  way  to  capture  knowledge  and  utilize  collective  intelligence  of  all  team 
members  is  to  use  prediction  markets  or  information  aggregation  markets.  Such  markets 
can  be  used  as  a  supplement  to  existing  estimation  methodologies  in  generating  more 
accurate  cost  and  schedule  estimates. 

Prediction  Markets:  History  and  Purpose 

Prediction  markets  are  emerging  as  a  promising  forecasting  mechanism  for 
efficiently  handling  the  dynamic  aggregation  of  dispersed  information  among  various  agents 
(Tziralis  &  Tatsiopoulos,  2007).  Various  terms  such  as  “information  markets,”  “decision 
markets,”  “electronic  markets,”  and  “virtual  markets”  can  be  used  to  describe  this.  In 
essence,  prediction  markets  are  speculative  markets  created  for  the  purpose  of  predicting 
the  outcome  of  an  uncertain  future  event.  They  provide  an  environment  for  traders  to  buy 
and  sell  contracts  whose  values  are  tied  to  uncertain  future  events. 

In  the  late  80s,  some  academics  at  the  University  of  Iowa’s  business  school  came  up 
with  idea  of  giving  students  hands-on  experience  in  trading  markets,  such  as  stock  and 
commodities.  Instead  of  using  play  money,  they  created  a  real  market  in  which  anyone 
could  bet  modest  sums  on  the  outcome  of  future  events — e.g.,  the  next  president  of  the 
United  States.  The  Iowa  Electronics  Markets  (IEM)  is  now  a  thriving  nonprofit  enterprise 
(Tetlock,  2006). 

Figure  1  shows  a  sample  prediction  market  that  is  commercially  available: 
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Boeing  7  87  -  When  will  the  Boeing  787  make  its  maiden  flight? 
Boeing  737  to-  make  maiden,  Flight  before  midnight  ET  on  31  Dec  20HQ-9 


Last  Price: 


47.0 


You  can  buy 
this  at  47.0 


Buy 


You  ca  n  sel  I 
this  at  3  6.5 


Sell 


Explain  |  |  Trade  Notv 


■  Closing  Price:  47 

Sep  OS  Oct 


Afghan  Politics  -  Who  will  become  President  of  Afghanistan  in  2.009  Presidential  Election? 
Hamid  Ka rxa i  to  be  sworn  in  as  President  of  Afghanistan  following  the  2QQ9  election 


Last  Price: 


so.o 


You  can  buy 
this  at  0.0 


Buy 


You  can  sell 
thisat  85.0 


Sell 


Explain 


Trade  Now 


Figure  1.  A  Sample  Prediction  Market 

(lntrade.com,  2010) 

The  theoretical  foundations  of  prediction  markets  lie  in  the  efficient  market 
hypothesis,  which  states  that  a  sufficient  number  of  marginal  traders  with  rational 
expectations,  who  maximize  utility  through  maximizing  profits,  drive  prices  in  the  market  in 
such  a  way  that  there  is  no  opportunity  for  arbitrage  (Shvarts  &  Green,  2007). 


It  is  believed  that  if  the  market  is  well  functioning,  then  contract  prices  reflect  the 
collective  wisdom  of  market  participants.  There  are  three  primary  types  of  prediction  markets 
(Wolfers  &  Zitzewitz,  2004): 


•  Winner  Takes  All  Market:  In  this,  the  contract  pays  a  specified  amount  if  and 
only  if  the  world  achieves  some  specific  conditions,  with  the  price  of  the 
contract  reflecting  the  market’s  assessment  of  the  probability  of  that 
occurrence. 


•  Index  Market:  In  this,  the  contract  pays  an  amount  reflected  in  some  condition 
of  the  world,  such  as  the  percentage  of  the  vote  a  candidate  receives  or  the 
inches  of  cumulative  snowfall  in  a  city. 

•  Spread  Market:  In  this,  contracts  specify  the  cutoff  that  determines  whether 
an  event  occurs  (such  as  win  margins  in  football).  These  basic  contract  types 
can  be  extended  to  generate  additional  predictions  and  measures  of  market 
uncertainty,  such  as  the  probability  that  a  candidate  will  receive  40%  of  the 
vote,  50%,  60%,  etc. 
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Table  1  illustrates  some  examples  for  each: 


Table  1.  Types  of  Prediction  Markets 

(adapted  from  Wolfers  &  Zitzewitz,  2004) 


Contract 


Example 


Details 


Reveals  market 
expectation  of ..  . 


Winner-take-all 


Event  y:  A1  Gore  wins  Contract  costs  $p. 

the  popular  vote.  Pays  $1  if  and  only  if 


Probability  that 

event  y  occurs,  p(y). 


event  y  occurs.  Bid 
according  to  value  of  $p. 


Index 


Contract  pays  $1  for  Contract  pays  $y. 

every  percentage 
point  of  the  popular 
vote  won  by  A1  Gore. 


Mean  value  of 
outcome  y:  E[y] . 


Spread 


Contract  pays  even  Contract  costs  $1.  Pays  $2 

money  if  Gore  wins  if  y  >  y*.  Pays  $0 

more  than  y*%  of  otherwise.  Bid  according 

the  popular  vote.  to  the  value  of  y*. 


Median  value  of  y. 


Prediction  markets  are  most  notably  used  today  for  predicting  election  outcomes, 
movie  box  office  returns,  terrorist  attacks  and  sporting  events  (Wolfers  &  Zitzewitz,  2004). 
Information  markets  have  also  been  deployed  at  Hewlett-Packard  Corporation  for  making 
sales  forecasts  and  were  found  to  be  more  effective  than  traditional  methods.  “Not  only  did 
the  market  predictions  consistently  beat  the  official  HP  forecasts;  the  outcomes  predicted 
are  consistent  with  the  probabilistic  predictions  of  the  1AM  (Information  Aggregation  Market)” 
(Plott  &  Chen,  2002). 

Google  also  launched  an  internal  prediction  market  in  April  2005.  The  Google 
markets  are  similar  to  Iowa  Electronic  Markets,  and  have  survey  questions  like  “How  many 
users  will  Gmail  have?”  Common  type  of  markets  included  those  forecasting  demand 
(Masse,  2008).  “Google’s  prediction  markets  are  reasonably  efficient,  but  did  exhibit  four 
specific  biases:  an  overpricing  of  favorites,  short  aversion,  optimism,  and  an  under-pricing  of 
extreme  outcomes.  New  employees  and  inexperienced  traders  appear  to  suffer  more  from 
these  biases,  and  as  market  participants  gained  experience  over  the  course  of  our  sample 
period,  the  biases  become  less  pronounced”  (Masse,  2008). 

Some  of  the  factors  that  are  involved  in  prediction  markets  are: 

•  Illiquidity:  Prediction  requires  enough  buyers  and  sellers  making  enough 
transactions  to  produce  a  clear  market  price  that  summarizes  the  market 
prediction.  In  “thin”  markets  where  buyer,  sellers  and  their  trades  are  few,  the 
prediction  market  can  sometimes  be  ineffective.  Some  research  suggests 
prediction  markets  can  be  effective  with  as  few  as  sixteen  active  traders 
(Christiansen,  2007). 

•  Rationality:  While  prediction  markets  have  been  strikingly  accurate  when 
they  meet  the  criteria  for  effective  markets,  scholars  have  noted  some 
deviations  from  “perfect”  rationality.  First,  while  traders’  preferences  may 
bias  their  trades,  such  as  when  partisans  buy  political  futures  contracts  for 
their  favored  candidate  (Forsythe,  Rietz  &  Ross,  1999),  the  prediction  market 
price  of  contracts  will  remain  accurate  so  long  as  the  enough  marginal  traders 
remain  objective  (e.g.,  their  profit  motives  outweigh  their  preferences). 
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Prediction  markets  do  not  require  all  traders  be  rational  as  long  as  marginal 
market  exchange  is  motivated  by  rational  traders  (Gruca  &  Berg,  2007; 

Tziralis  &  Tatsiopoulos,  2007).  Second,  as  in  horse  racing,  prediction  markets 
participants  exhibit  over  price  long  shots  (so  that  contract  holders  receives  a 
smaller  payoff  relative  to  the  true  probability  of  the  event  occurring)  and  under 
price  high  probability  events,  though  Cowgill,  Wolfers,  and  Zitzewitz  (2009) 
suggest  that  precisely  the  opposite  biases  occurred  when  Google  employees 
traded  in  internal  company  prediction  markets  forecasting  Google’s  future. 
Moreover,  prediction  markets  may  be  less  effective  at  accurately  predicting 
extremely  likely  or  unlikely  events  (Wolfers  &  Zitzewitz,  2004). 

•  Manipulation  and  Bubbles:  A  sufficiently  endowed  and  motivated  trader 
might  attempt  to  manipulate  a  prediction  market  by  purchasing  contracts  in 
desired  directions.  A  candidate  for  elected  office,  for  example,  might 
purchase  contracts  on  himself  to  generate  apparent  momentum  and  publicity. 
Speculative  bubbles  are  possible  in  prediction  markets,  a  non-trivial  concern 
in  light  of  recent  macroeconomic  events.  Wolfers  and  Zitzewitz  (2004) 
propose  that  a  speculative  bubble  occurred  in  political  prediction  markets  on 
whether  Hillary  Clinton  would  win  the  2004  Democratic  presidential  primary, 
and  bubbles  may  have  occurred  in  experimental  prediction  markets  reported 
in  Plott  and  Sunder  (1988).  To  some  extent,  markets  may  self  correct 
manipulation  and  bubbles  (Strumpf,  2004).  The  best  defense  against 
manipulation  is  sufficient  liquidity  so  that  profit  seeking  traders,  recognizing 
that  the  manipulated  market  prices  are  inaccurate  trade  for  contracts  to  bring 
predictions  back  into  line. 

Prediction  Markets  for  Defense  Acquisition  Projects 

The  DoD’s  ability  to  meet  its  acquisition  targets  is  becoming  increasingly  critical  as 
defense  budgets  are  reduced  and  expectations  to  deliver  on  time  and  on  budget  remain 
high.  Costs  overruns  are  one  aspect  of  this  issue.  Please  refer  to  Table  5  in  the  Appendix 
for  a  sample  of  cost  overruns  in  large  federal  projects. 

A  recent  GAO  report  (Edwards,  2009)  concluded  that  “weapon  programs  are  taking 
longer,  costing  more  and  delivering  fewer  capabilities  than  originally  planned.”  It  also  noted 
that  “systematic  problems  both  at  the  strategic  and  at  the  program  level”  (GAO,  2008)  were 
to  blame.  The  GAO  also  noted  that  military  branches  “overpromise  capabilities  and 
underestimate  costs  to  capture  the  funding  needed  to  start  and  sustain  development 
programs”  (GAO,  2008). 

In  2008,  the  GAO  reviewed  cost  and  schedule  of  72  weapons  programs  and  found 
that  the  average  cost  overrun  for  systems  development  was  40%  (GAO,  2008)  and 
concluded  that  “DOD’s  acquisition  outcomes  appear  increasingly  suboptimal”  (GAO,  2008). 

A  study  by  Deloitte  consulting  also  agrees  that  defense  cost  overruns  are  getting  worse 
(Irwin,  2009). 

Table  2  gives  a  brief  comparison  between  existing  cost  estimation  techniques  and 
the  prediction  markets.  The  goal  here  is  to  compare  the  probable  shortcomings  of  the  former 
with  the  possible  advantages  of  the  latter. 
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Table  2.  Value  Propositions  of  Prediction  Markets 


Probable  limitations  of  existing  cost 
methods 

Possible  value  propositions  of  prediction 
markets 

•  Cost  Estimating  Relationships  (CERs) 
based  on  technical  factors  rather  than 
programmatic  “soft”  factors. 

•  Shift  of  focus  from  estimating  by 
individuals  to  groups. 

•  Not  dynamically  updated  as  a  program 
evolves,  making  the  original  estimate 
outdated  as  soon  as  the  climate 
changes. 

•  Provide  way  to  leverage  information  from 
diverse  sources. 

•  Manifestation  of  a  few  decision-makers, 
under  tremendous  time  pressure, 
working  with  limited  and  perhaps  biased 
information. 

•  Mitigate  biases  stemming  from  pressure 
to  “price  to  win”  or  hide  damaging 
information. 

•  Enable  freguent  sampling  of  information. 

•  Incentivize  traders  to  seek  out  gualitv 
information  that  will  help  them  do  better. 

Prediction  markets  are  a  judgmental  forecasting  method  (Graefe,  2009)  in  which  the 
price  mechanism  of  the  market  automatically  aggregates  all  the  dispersed  information 
among  participants.  Various  aspects  of  the  prediction  markets  are  also  discussed  in  further 
sections. 

Design  Principles  of  Prediction  Markets 

As  with  any  markets,  prediction  markets  may  fail — and  produce  inaccurate 
forecasts — if  not  properly  designed  and  executed.  They  should  follow  specific  mechanisms, 
contracts  and  contexts  to  ensure  participants  are  adequately  incentivized  to  participate. 

Some  other  factors  to  be  incorporated  are  (Suda,  2009): 

•  Should  require  unbiased  tacit  knowledge  of  crowd, 

•  Should  be  designed  in  well-incentivized  environments, 

•  Should  be  designed  for  a  non-hierarchical  environment, 

•  Generally  not  suitable  for  idea  generation  process, 

•  Should  have  well-defined  end  dates  or  closing  criteria,  and 

•  Choices  should  be  mutually  exclusive  and  have  a  definitive  outcome. 

Pilot  Studies  on  Prediction  Markets 

In  order  to  study  the  prediction  markets,  we  conducted  some  simulations  on  a  group 
of  students  at  MIT  to  study  the  effectiveness  of  prediction  markets,  and  some  interesting 
observations  were  noted.  Details  of  the  study  are  given  below. 

Pilot  Study  1:  Predicting  Ratings  for  Sunday  Night  Football 

A  group  of  17  participants  (N=17)  were  asked  to  participate  in  a  private  marketplace 
for  predicting  the  Nielsen  Rating  (NBC  -  USA)  for  Sunday  Night  Football  (Oct  25).  Please 
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refer  to  Figures  8,  9,  and  10  in  the  Appendix  for  the  screenshots  of  the  market.  The  following 
rules  were  set  in  place: 

•  Time  duration  for  trading  was  5  days,  with  the  market  set  to  close  30  minutes 
prior  to  game  start. 

•  Each  participant  was  allotted  an  initial  amount  of  $5,000  for  trading  (play 
money). 

•  Winning  Criteria  was  generating  maximum  worth. 

•  Incentive — Points  for  class  participation  grade  and  a  popular  book  for  winner. 

•  Minimum  of  four  trades  per  person  (tied  to  the  incentive  of  class  grade),  with 
no  maximum  limit  on  trading. 

•  1 1  rating  ranges  were  presented  as  choices  or  contracts  for  participants  to 
trade  on,  with  each  rating  range  being  assigned  equal  probability  (9.09% 
each)  to  start  with  (See  Figure  2). 

•  Participants  could  “buy”  or  “sell”  shares  of  the  choice  or  contract  they  found 
most  likely  to  occur.  Every  time  a  user  bought  a  share  for  a  particular  idea, 
the  price  went  up.  Similarly,  each  time  users  sold  a  share,  the  price  went 
down.  The  user’s  account  was  also  credited  or  debited  based  on  his/her 
choice. 

•  There  was  a  direct  correlation  between  price  and  probability — share  price  of 
$9.09  indicated  9.09%  chance  of  that  particular  rating  range. 

•  The  buy  and  sell  transactions  were  managed  by  an  automatic  market  maker. 

Results 


Around  236  trades  were  conducted  in  total  by  17  participants. 

The  highest  rating  range  predicted  was  of  (9.6-10)  with  a  19%  likelihood  prediction, 
followed  by  range  (9. 1-9.5)  with  a  17.3%  chance  (see  Figure  3). 

What  will  be  the  Nielsen  Ratings  for  the  NBC  Sunday  Football 
Game  (Oct25j? 
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Figure  2.  The  Stocks  in  the  Prediction  Market 

(lnklingmarkets.com) 
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The  actual  rating  range  turned  out  to  be  (9. 1-9.5). 

Actual  Range 

Predicted  Range  (9. 1-9. 5) 
(9.6-10.0)  a 


Figure  3.  Results  at  the  Close  of  Prediction  Market 

A  probability  distribution  function  (Pdf)  for  the  result  is  shown  in  Figure  4,  which 
demonstrates  how  close  the  predicted  result  was  to  the  actual  result. 


Although  market  predicted  range  was  close,  there  can  be  two  reasons  why  the 
market  forecast  was  not  absolutely  accurate: 
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On  the  day  of  the  game,  a  World  Series  game  ran  into  overtime — overlapping 
with  the  football  game  and  impacting  the  expected  ratings  for  the  Sunday 
night  football.  It  turns  out  that  there  were  no  baseball  fans  in  the  participating 
traders,  and  hence  this  information  was  not  captured. 
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•  In  a  study  on  markets  on  NFL  football  games  from  tradesports.com,  there 
was  an  observation  of  mispricing  (Solomon  &  Hartzmark,  2008)  visible 
primarily  in  sporting  markets.  This  was  due  to  a  behavioral  bias  known  as 
disposition  effect,  which  is  the  tendency  of  investors  to  sell  stocks  that  have 
risen  in  value  and  hold  on  to  stocks  that  have  fallen  in  value,  with  the 
objective  of  earning  maximum  profit.  Thus  there  might  be  mispricing  in  such 
scenarios  and  the  hypothesis  that  the  prices  in  prediction  markets  reveal 
unbiased  estimates  of  true  probabilities  of  the  event  happening  might  not  be 
true. 

Pilot  Study  2:  Predicting  Ratings  for  Monday  Night  Football 

Based  on  the  previous  pilot  study  feedback,  the  structure  of  the  next  market  was 
modified.  The  participants  (N=13)  were  asked  to  predict  the  Nielsen  Rating  (Cable  TV  -  US) 
for  Monday  Night  Football  on  ESPN.  Instead  of  providing  range  choices  to  participants  for 
bidding,  they  were  asked  to  bid  on  an  indexed  market.  Rules  were  similar  to  the  last  market, 
except  for  the  following  changes: 

•  Instead  of  ranges,  participants  now  saw  only  one  rating.  They  had  to  predict  if 
the  rating  as  per  their  opinion  was  to  be  higher  or  lower.  Based  on  their 
responses,  the  price  or  rating  was  adjusted  automatically. 

•  The  initial  rating  set  was  10. 

•  The  price  and  rating  scale  was  1:1,  i.e.,  a  price  of  $10.00  meant  the  current 
projected  value  of  rating  was  10.00  (see  Figure  5). 

Results 

Around  1 16  trades  were  conducted  by  13  participants.  The  final  rating  predicted  by 
the  market  was  10.00  (see  Figure  6). 

What  will  be  the  Nielsen  Rating  (Cable  TV  -  US)  for 
Monday  Night  Football  on  ESPN  (Nov 2)? 


Rating  $10.00 
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Nielsen  Ratings 


Figure  6.  Graph  Showing  Change  in  Value  of  the  Stock  upon  Trading 

There  was  a  huge  variation  in  the  ratings  predicted,  with  the  highest  value  being  33.1 
and  the  lowest  value  being  0.6. 

The  actual  rating  turned  out  to  be  7.7,  which  was  different  from  the  market  value. 

The  major  reason  for  this  was  that  market  place  was  too  volatile.  A  single  participant  could 
buy  a  large  number  of  shares  and  alter  the  rating  to  a  great  extent,  which  is  not  a  good 
design  principle  for  prediction  market. 


Observations 

Behavioral  Observations  of  Participants 

The  participants  comprised  a  mix  of  people  with  football  knowledge  and  others  with 
no  knowledge  at  all.  The  actual  result  of  the  rating  was  quite  close  to  the  rating  range  that 
the  test  market  predicted.  But  the  main  benefit  was  observed  in  the  behavior  of  the 
participants.  An  unbiased,  incentivized  market  piqued  their  interest  and  motivated  them  to 
remain  involved  throughout  the  market  duration.  Some  other  observations  were: 

•  Increased  participation  due  to  internal  competition, 

•  Attempts  at  market  gaming, 

•  Trying  to  make  use  of  First  mover  advantages, 

•  Forming  alliances  to  manipulate  the  market, 

•  High  incentive  to  participate  led  participants  to  gain  more  knowledge  on  the 
subject,  and 

•  Use  of  statistical  models  by  some  participants  to  forecast  ratings. 

Other  Useful  Insights  from  Pilot  Studies  for  Implementing  Prediction 

Markets 

Ensure  enough  liquidity.  Research  done  on  prediction  markets  (Christiansen, 
2007)  suggests  at  least  16  participants  for  an  effective  market.  Based  on  the 
two  pilot  studies,  the  number  of  participants  was  lesser  in  the  second  study, 
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potentially  explaining  the  volatility  and  resulting  significant  error  in  the 
prediction. 

•  Avoid  volatility  by  choosing  the  right  scale  for  trading  in  market.  The  market 
needs  to  be  less  volatile,  so  that  it  is  not  easily  influenced  by  a  single  trader. 

•  Ease  of  use.  The  prediction  market  should  be  easy  to  use,  as  many 
participants  were  initially  reluctant  to  trade  due  to  perceived  complexity. 

•  Ensure  regular  information  update.  In  both  the  studies,  there  were  not 
enough  “happenings”  regarding  the  game  ratings,  during  the  trading  duration 
to  sustain  participants’  interest.  The  market  is  expected  to  fare  even  better  is 
some  new  knowledge  is  available  throughout  the  trading  duration. 

•  Leads  to  increased  awareness  and  interest.  A  significant  contribution  of 
prediction  markets  in  the  field  of  cost  or  schedule  estimation  is  to  keep  the 
participants  involved  and  make  the  process  exciting.  Traders  with  no 
knowledge  of  football  got  involved  by  researching  more  on  the  subject. 

•  Align  incentives  with  the  participants’  interests.  Some  people  just  executed 
the  minimum  required  four  trades  and  stopped.  Although  there  was  high 
trading,  some  people  did  not  care  to  do  all  the  research.  One  reason  for  this 
was  due  to  incentives  not  aligning  directly  with  the  objectives  of  these 
participants. 

Designing  Prediction  Markets  for  Defense  Acquisition  Projects 

There  are  various  components  of  a  market  that  need  to  be  considered  prior  to 
designing  and  implementing  a  prediction  market  for  a  project.  A  prediction  market  is 
composed  of  the  following: 

Stocks:  These  are  the  outcomes  or  possibilities  of  the  market  that  are 
collectively  exhaustive  and  mutually  exclusive.  These  act  as  stocks  and 
securities  for  the  traders  to  trade  against,  with  a  “buy”  indicating  a  belief  of 
occurrence  of  that  particular  outcome  and  a  “sell”  indicating  a  belief  of  non¬ 
occurrence. 

Marketplace:  This  refers  to  the  software  environment  that  contains  the 
market  or  the  questions  for  trading.  It  also  encompasses  the  market  manager 
that  facilitates  trades  among  the  traders  and  manages  the  information 
distribution  process. 

Traders:  These  are  the  participants  or  team  members  in  a  project  that  will  be 
trading  information  in  the  form  of  stocks  in  the  market. 
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Figure  7.  The  Composition  of  the  Prediction  Marke 

There  needs  to  be  a  continuous  flow  of  information  for  the  traders  to  base  their 
decisions  on  and  trade  throughout  the  market  duration. 

Incentive  structure  is  another  important  factor  that  should  be  considered  during 
market  design,  as  it  acts  as  the  motivating  factor  for  the  traders. 

The  success  of  the  project  hinges  on  identifying  an  acquisition  program  willing  to 
participate.  The  ideal  characteristics  of  the  program  are: 

•  uncertain  cost  and  schedule  components  that  can  be  specified  clearly  in  a 
contract, 

•  a  sufficiently  large  program  community  to  ensure  a  liquid  market, 

•  ample  “soft”  relative  to  “hard”  information  and  information  is  broadly  and 
unevenly  held  by  diverse  actors, 

•  one-of-a-kind  program  or  a  program  with  limited  relevant  historical 
information,  and 

•  susceptible  to  performance  impacts  as  a  result  of  external  events  (i.e., 
political  landscape,  policy  changes,  personnel  attrition,  technology  maturity, 
design  modifications). 

Based  on  initial  pre-screening,  we  identified  “Shallow  Water  Combat  Submersible” 
(SWCS)  as  a  suitable  project,  with  a  plan  to  implement  the  prediction  market  for  duration  of 
3-6  months. 

Below  is  some  of  the  discussion  on  the  design  components  for  the  prediction  market. 

1 .  Stocks:  The  stocks  or  the  outcomes  traded  in  the  prediction  market  form  the 
most  important  component  of  the  market.  The  stocks  traded  will  give  us  an 
indication  of  the  beliefs  of  the  participants,  and  the  number  of  stocks  traded 
will  show  us  the  level  of  confidence  in  their  beliefs. 

Hence,  the  questions  that  are  asked  in  the  market  are  very  important. 

There  are  mainly  two  categories  of  questions  that  can  be  asked. 

Type  I  questions:  Asking  these  questions  can  directly  provide  the  required 
information.  Qi  ->A 

Type  II  questions:  Asking  these  questions  can  indirectly  provide  the  required 
information,  and  a  series  of  questions  might  be  required  for  getting  the 
information  directly.  Q2  Q’  Q”  ->  A 
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This  is  similar  to  another  perspective,  as  per  which  the  state  of  the  world  can  be 
divided  on  basis  of  two  dimensions  (Healy,  Ledyard,  Linardi  &  Lowery,  2009).  Some 
questions  can  based  on  an  unobservable  factor  whose  value  can  impact  realization  in 
second  dimension,  e.g.,  the  underlying  monetary  policy  of  a  central  bank  .While  some 
questions  can  be  based  on  a  directly  observable  factor,  e.g.,  interest  rate  each  quarter. 

The  latter  type  of  questions  correspond  to  Type  I  questions,  while  the  former 
compare  to  Type  II  questions. 

Handling  Type  II  Questions 

The  distinction  of  questions  based  on  these  categories  is  important  because  these 
allow  us  to  focus  more  carefully  on  the  “stocks”  or  outcomes  of  Type  II  questions,  so  that  the 
required  information  is  available. 

•  The  suitable  approach  with  the  Type  II  questions  is  to  ask  multiple  questions 
in  the  first  round  of  prediction  market,  and,  subsequently,  drill  down  to 
specific  questions  in  further  rounds. 

•  If  we  consider  a  single  question  as  a  single  market,  then  one  of  the 
approaches  can  be  to  run  multiple  markets  at  a  given  time,  and  then  to  add 
follow-up  questions  based  on  the  responses.  New  markets  (i.e.,  questions) 
can  be  run  every  week,  and  old  questions  can  be  updated  weekly  based  on 
new  information.  The  important  factor  with  this  approach  is  to  decide  which 
questions  need  to  be  deployed  first. 

Designing  Questions  for  the  Market 

There  is  a  lot  of  information  that  can  prove  useful  in  case  of  a  defense  acquisition 
project.  This  information  can  be  either  related  to  any  factors  impacting  the  cost  or  schedule 
of  the  project,  or  can  be  related  to  the  current  or  future  decision  processes  within  the  project. 
Based  on  the  type  of  information  required,  we  have  further  categorized  the  market 
questions. 

•  Questions  of  the  First  Order:  These  are  the  questions  that  give  us  the 
information  about  the  program  of  interest.  Some  examples  of  questions  for 
the  pilot  project  are: 

Table  1.  Will  SWCS  be  certified  by  August  1, 2010? 

Table  2.  The  cost  of  the  first  unit  will  be  $x. 

Table  3.  The  final  cost  of  the  program  will  exceed  the  baseline  estimate  by  x%. 

•  Questions  of  the  Second  Order:  These  are  the  questions  that  give  us  the 
information  about  traders  and  the  trading  process.  For  the  pilot  project,  some 
of  the  questions  are: 

Table  4.  What  is  your  motivation  for  trading  (e.g.,  to  win/to  solve  the  problem/to 
validate  my  knowledge)? 

Table  5.  What  is  your  role  in  the  Organization? 

Table  6.  What  are  useful  sources  of  information  about  the  project? 
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•  Questions  of  the  Third  Order:  These  questions  give  us  information  of  the 
behavior  outside  of  the  prediction  markets.  Some  of  the  questions  are: 

Table  7.  How  have  you  used  the  prediction  market  in  your  job?  If  you  notice  a 
change  in  a  contract  price,  how  have  you  used  that  information? 

Table  8.  Did  trading  improve  your  confidence  in  your  opinion? 

The  First  Order  questions  will  be  asked  in  the  prediction  market  for  trading.  The 
Second  Order  and  Third  Order  questions  are  more  of  profiling  questions,  which  can  help 
decide  how  to  improve  the  markets,  or  how  the  participants  are  gaining  from  the  market 
information.  These  questions  can  be  sent  to  the  participants  as  a  survey  and  can  be  tied  to 
incentives  so  as  to  encourage  them  to  participate.  Incentive  can  be  additional  bonus  money 
(like,  Earn  $100  more  (of  play  money)  for  answering  5  questions) 

The  above  factors  helped  to  shortlist  some  of  the  questions  to  be  used  in  the 
prediction  market  for  the  pilot  defense  project  (SWCS).  A  risk  mapping,  showing  the  risk 
covered  by  a  specific  question  of  the  First  Order,  along  with  the  modified  version  of  the 
question,  is  shown  below.  Note  that  the  first  four  questions  are  of  Type  I,  while  the  last 
question  is  of  Type  II. 

Table  3.  Risk  Mapping  for  First  Order  Questions 


Question 

Type 

First  Order  Questions 
(Information  about  the  program  of 
interest) 

Measured  Risk 

Alternate  way  of  askingthe  question 

Type  1 

Will  SWCS  be  certified  by  August  1,  2010? 

Schedule  Overrun 

When  will  SWCS  be  certified? 

By  how  many  days  will  the  certification  date  be 
exceeded? 

What  do  you  think  is  the  most  significant  cost  driver 
for  the  first  Unit? 

Type  1 

The  cost  of  the  first  unit  will  be  $x. 

Cost  Overrun 

Follow  up  question: 

Based  on  the  below  ranking  of  cost  drivers  for  the 
first  Unit,  what  do  you  think  the  cost  of  first  Unit  will 
be? 

Type  1 

The  final  cost  of  the  program  will  exceed 
the  baseline  estimate  by  x%. 

Cost  Overrun 

Type  1 

What  will  be  the  2011  Fiscal  Year 
appropriation  for  the  SWCS? 

Costand  Schedule 

Type  II 

The  SWCS  program  will  be  adequately 
resourced  to  meet  its  milestones? 

Resource 

Allocation  Errors 

By  how  many  days/months  will  the  milestone  be 
missed? 

Which  resource  do  you  think  is  most  necessary  to 
meet  the  program's  schedule  objective? 

Given  the  available  resources,  will  the  project  be  able 
to  meet  its  planned  schedule? 

Another  important  aspect  is  to  mix  some  fun  questions  (like,  who  will  win  the  Monday 
night  game,  etc.)  along  with  serious  questions  to  keep  the  participants  involved. 

•  Marketplace:  There  are  two  important  aspects  of  design  for  the 

marketplace — the  software  environment  in  which  the  participants  perform 
trades  and  the  market  manager  who  facilitates  the  trading  process. 
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Considerations  for  the  Software  Environment 

The  following  are  the  design  considerations  for  the  software  environment,  based  on 
observed  behaviors  in  pilot  simulations  and  requirements  for  a  defense  environment: 

•  The  software  environment  will  be  easy  to  access  and  use. 

•  The  software  environment  will  be  commercially  available,  with  prior  success 
in  defense  related  environment. 

•  The  environment  will  satisfy  the  IT  security  considerations — like  secure  login, 
user  access  rights,  data  security,  etc. 

•  The  software  environment  will  have  dashboard  and  administrative  capabilities 
to  analyze  the  data  and  generate  reports  based  on  the  user  behavior. 

•  It  will  be  possible  to  share  new  information  with  the  traders  and/or  close  or 
open  new  markets  whenever  required. 

Considerations  for  the  Market  Manager 

The  matching  mechanism  in  prediction  markets  has  most  often  been  a  continuous 
double  auction  in  which  computer  software  matches  buyers  offering  bids  with  sellers  and 
their  asking  prices.  Prediction  markets  have  been  successful  using  real-world  money,  purely 
“play  money,”  with  no  economic  value  (beyond  the  satisfaction  of  “winning),  and  “prize 
money”  that  can  be  exchanged  for  prizes  and  entries  in  prize  drawings.  We  will  establish 
prediction  markets  with  play  and  prize  money  and  will  establish  contracts  for  the  cost  and 
duration  of  particular  features  of  the  acquisition  program.  This  is  more  realistic  than  focusing 
on  the  entire  program  completion,  given  the  limited  time  we  have  to  run  the  market.  Fully 
specified,  clear  and  enforceable  contracts  in  prediction  markets  require  the  future  state  of 
the  world  (or  events)  on  which  the  contracts  can  be  easily  adjudicated. 

Market  Scoring  Rule,  invented  by  Professor  Robin  Hanson  at  George  Mason 
University,  will  be  used  in  the  prediction  market.  This  will  ensure  that  fewer  traders  can  be 
present  in  the  market,  and  lot  of  concepts  related  to  stock  markets  that  are  difficult  to 
understand  (like  bid-ask  spreads,  etc.)  can  be  avoided  (Siegel,  2010).  The  price  of  the  stock 
represents  whatever  the  last  trader  was  willing  to  pay  for  shares  and  is  set  automatically 
according  to  the  volume  and  sentiment  in  the  question.  For  example,  if  someone  buys  10 
shares  of  an  answer,  the  price  of  the  share  will  be  increased  automatically  according  to  an 
algorithm. 

To  ensure  sufficient  liquidity,  we  will  use  a  prediction  market  with  automated  market 
maker,  which  means  that  the  buying  and  selling  of  stocks  will  happen  between  traders  and 
the  market  and  not  between  the  traders  themselves. 


•  Traders:  While  prediction  markets  require  some  disagreement  among 
potential  traders  over  forecast,  (else  no  trades  would  occur)  excessive 
information  heterogeneity  can  be  harmful  to  the  markets  functioning.  If  some 
traders  posses  significant  private  information  (and  hence  become  insiders), 
the  outsiders  may  refuse  to  participate,  ultimately  killing  the  market. 
Prediction  markets  are  most  likely  to  be  successful  if  traders  hold  sufficiently 
balanced  information  about  the  event  but  have  differing  beliefs  about  the 
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meaning  of  common  information.  It  is  also  critical  that  when  new  information 
becomes  available,  it  will  be  widely  known. 

We  will  recruit  market  participants  from  those  employed  in  the  Acquisition  program  or 
those  surrounding  the  Acquisition  program  (i.e.,  subcontractors,  end  users,  consultants). 

The  traders  in  the  defense  project  comprise  of  the  different  team  members  across  different 
cross-functional  areas.  There  will  be  many  people  in  a  project  who  will  have  limited 
information  as  to  what  is  going  on  with  a  program  at  large,  and  they  might  be  biased  to  their 
individual  projects.  Hence,  it  will  be  ensured  that  participants  from  all  smaller  projects,  as 
well  as  people  from  other  departments  are  included  in  the  trading  process.  The  acquisition 
project  will  have  sufficiently  broad  following  to  attract  participants  from  inside  and  outside 
the  government.  These  can  include  administrative  personnel,  sub-contractors,  engineers, 
end  users,  consultants,  etc.  The  markets  will  be  implemented  for  duration  of  3  to  6  months, 
giving  ample  trading  time  to  the  participants. 

Sometimes,  due  to  fear  of  hierarchy  or  work  environment  pressures,  people  do  not 
reveal  uncomfortable  information  related  to  the  project.  In  the  prediction  market,  anonymity 
of  traders  will  be  maintained  so  that  they  can  disclose  useful  information  without  hesitation. 


•  Incentives:  The  incentive  structure  is  perhaps  the  most  important  component 
of  prediction  market  in  the  DoD  environment.  Incentives  should  be  such  that 
they  are  non-biased  and  encourage  participation  independent  of  any 
hierarchical  considerations.  “As  with  any  business  incentive  system,  a 
considerable  challenge  exists  in  choosing  incentives  that  motivate  the  right 
behavior.  . .  With  information  markets,  incentives  must  serve  a  dual  role:  to 
motivate  participation  and  to  motivate  participants  to  provide  truth-revealing 
opinions.  Incentives  that  satisfy  both  criteria  can  be  difficult  to  define” 
(LaComb,  Barnett  &  Pan,  2007). 

Performance-based  incentives  in  the  project  can  be  provided  in  various  ways.  For 
example,  participants  can  be  rewarded  based  on  the  portfolio  value  at  the  closure  of  the 
market.  Awards  can  also  be  given  to  the  most  active  traders,  or  the  traders  contributing  the 
most  valuable  information. 

Non-monetary  awards  can  be  as  effective  as  monetary  awards  when  used  as 
incentives.  For  example,  Google  found  that  in  their  internal  markets,  participants  cared  more 
about  non-cash  prizes  like  t-shirts  rather  than  cash  prizes  (Coles,  Lakhani  &  McAfee,  2007). 
Similarly,  by  announcing  user  rankings,  the  play-money  markets  in  Christiansen’s  field 
experiment  (2007)  were  successful,  even  without  monetary  incentives. 

Other  factors  like  social  competition,  recognition  and  opportunity  to  contribute 
towards  the  project  can  also  act  as  important  motivating  factors. 

For  the  pilot  defense  acquisition  project,  we  feel  the  following  incentives  will 
encourage  participation: 

Declaring  winners  based  on  the  maximum  portfolio  value  at  market  closure, 

Maintaining  a  leadership  board  displaying  the  top  players,  and 

Including  results  of  the  prediction  market  in  regular  status  reports  to  track  any 
new  or  less  visible  information. 
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Conclusion 


In  his  book  The  Wisdom  of  Crowds,  James  Surowiecki  (2005)  argued  that  if  we  take 
a  crowd  of  diverse  people  and  correctly  aggregate  their  judgments,  we  will  be  able  to  get 
more  accurate  results  based  on  the  collective  intelligence  of  the  crowd.  This  wisdom  of 
crowds  can  be  seen  in  action  every  day  (Coles,  Lakhani  &  McAfee,  2007)  with  the  collective 
intelligence  also  driving  Wall  Street — the  probabilities  generated  from  the  market  displayed 
through  the  stock  prices. 

The  prediction  markets  can  be  implemented  for  defense  acquisition  projects  to 
aggregate  the  dispersed  information  across  different  functional  areas.  This  information  can 
help  keep  track  of  any  cost  or  schedule  variances,  and  also  identify  any  potential  risks  that 
can  impact  the  project.  We  anticipate  prediction  markets  can  outperform  existing  Defense 
Acquisition  estimation  techniques  (i.e.,  parametric,  analogy,  activity-based  (Boehm,  Valerdi, 
Lane  &  Brown,  2005)  in  cases  where  there  is  ample  “soft”  relative  to  “hard”  information  and 
information  is  broadly  and  unevenly  held  by  diverse  actors. 

Markets  need  to  be  compatible  with  the  political  and  regulatory  contexts  in  which 
they  operate.  For  example,  federal  regulations  permit  play  and  prize  money  prediction 
markets  but  have  restricted  real  money  markets.  Perhaps  the  most  notorious  prediction 
market  was  the  ill-fated  DARPA  sponsored  policy  prediction  markets  (Hanson,  2007),  which 
collapsed  from  a  political  firestorm  over  the  (quite  illogical)  proposition  that  terrorists  might 
benefit  from  trading  in  terrorism  futures. 

Based  on  the  various  case  studies  and  literature  available  on  prediction  markets,  our 
observations  through  simulations  of  markets,  and  discussion  on  various  aspects  of  defense 
acquisition  projects,  we  consider  the  following  design  principles: 

•  The  acquisition  project  should  have  a  sufficiently  broad  following  to  attract 
participants  from  inside  and  outside  the  government.  These  can  include  sub¬ 
contractors,  end  users,  consultants,  etc. 

•  Although  play  money  should  be  used  for  participation  in  the  markets,  the 
traders  can  be  incentivized  based  on  their  constant  participation  and  ability  to 
predict  accurate  results. 

•  Ample  historical  data  should  be  available  on  similar  projects,  so  that 
participants  who  are  new  to  the  project  have  some  basis  of  gaining 
knowledge  and  predicting. 

•  The  questions  for  the  market  should  be  very  carefully  designed  so  that  the 
required  information  can  be  revealed.  “Fun”  questions  should  be  mixed  with 
“hard”  questions  so  that  the  participants  remain  involved. 

•  To  ensure  sufficient  liquidity,  prediction  markets  with  automated  market 
makers  should  be  used,  and  at  least  15-16  participants  should  be  targeted  for 
trading  (Christiansen,  2007). 

•  Results  of  the  prediction  markets  should  be  carefully  studied  and  reported  as 
part  of  regular  status  reports. 

•  Anonymity  of  participants  with  respect  to  responses  should  be  maintained  to 
encourage  information-sharing. 
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•  The  length  of  the  market  and  closure  of  each  specific  question  need  to  be 
given  proper  consideration  while  designing  the  market.  The  support  of  senior 
management  in  Defense  projects  is  essential  for  starting  the  prediction 
market. 

•  Markets  need  to  be  compatible  with  the  political  and  regulatory  contexts  in 
which  they  operate. 

As  part  of  the  ongoing  research  project,  the  observations  and  feedback  from 
preliminary  and  literature  research  will  be  used  to  run  the  field  trial  with  the  DoD 
(Department  of  Defense)  acquisition  programs  for  a  fixed  duration  (up  to  6  months).  The 
prediction  market  for  the  selected  pilot  acquisition  project  still  needs  to  be  implemented,  and 
other  design  and  implementation  factors  will  be  added  or  the  current  ones  revised  based  on 
new  information. 

If  prediction  markets  improve  in  forecasts  in  the  defense  arena,  as  they  do  in  other 
venues,  our  aim  will  be  to  assist  in  implementing  and  then  studying  additional  markets, 
including  those  over  the  longer  term  and  a  wider  variety  of  programs.  It  is  clear  from  our 
research  that  a  significant  contribution  of  prediction  markets  in  the  field  of  cost  or  schedule 
estimation  is  to  keep  the  participants  involved  and  make  the  process  exciting.  Used  with 
right  implementation  guidelines,  these  markets  can  prove  to  be  helpful  in  capturing  the 
collective  wisdom  of  the  project  members  for  the  specific  duration. 
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Appendix 


Table  4.  Development  Cost  Overruns  by  Decade 

(GAO,  2006) 


1970-1979 

.1- .  .Ml- 

Development 
cost  overrun: 

$13  billion 

(30%) 

Development 
cost  overrun: 

$12  billion 

(39%) 

Development 
cost  overrun: 

$15  billion 

(40%) 

Key  Studies  and  Initiatives  Impacting  the  Defense  Acquisition  Process 

•  1 970  Fitzhugh  Commission 

•  1 972  Commission  on  Government 
Procurement 

•  1981  Carlocci  Initiatives 

•  1 982  Grace  Commission 

1 986  Packard  Commission 

•  1 994  Federal  Acquisition  Streamlining  Act 

•  1 996  Clinger-Cohen  Act 

DOD  Acquisition  Policy  Changes 

#  1 971  DOD  5000  policy  established 

#  1 975  Policy  revised 

#  1 977  Policy  revised 

•  1980  Policy  revised 

•  1 982  Policy  revised 

•  1 985  Policy  revised 

•  1 986  Policy  revised 

•  1987  Policy  revised 

•  1 991  Policy  revised 

•  1 996  Policy  revised 

Source:  DOO  (data):  GAO  (analysts  and  presentation). 


Table  5.  Sampling  of  Federal  Cost  Overruns 

(Edwards,  2009) 


Defense  Development 

Costs  ($2008 j13 

Original  Estimates 

Finai  Estimates 

Global  Hawk  surveillance  plane 

5989m  (2001) 

43.7b  [2007) 

Expeditionary  Fighting  Vehicle 

Sl.6b  (2000) 

S3 .6b  [2007) 

C-iiOJ  Hercules 

109m  (1996) 

430.3m  [2007) 

Extended  flange  Munitions 

8$, 9m  (1997) 

500,  im  (2007) 

DDG  1000  destroyer 

2.2b  [1946) 

9.3b  (2006) 

V-J2  Osprey  helicopter 

4. Oh  [1966) 

12.5b  [2006) 

Armed  Reconnaissance 

Helicopter 

386.3m  (200$) 

750.9  (2007) 

Space  Based  Infrared  System 
High 

4.2b  [1996) 

ar5t>  (2006) 

Armed  Reconnaissance 
Helicopter 

366.3m  (2003) 

750.6  (2007) 

NP0€SS  Satellite  System 

5.0b  [2002) 

7.9b  (2007) 

Other  Defense 

Coastal  Patrol  Ships'^ 

5220m  (2004) 

5350m  (2007) 

Joint  Strike  Fighter65 

5232b  (2001) 

532? D  (2008) 

Marine  one  (VH-7i) 
helicopters*6 

56.  lb  (2005) 

S!l_2b  (2008) 

Coast  Guard,  NSC  ships,  per 
■unit67 

5290m  (2002) 

5536m  (2007) 

m  XltoTiAM 


w 
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What  will  be  the  Nielsen  Ratings  for  the  NBC  Sunday 

Football  Game  (Oct  25)? 

■  a- 

■  ith  ■  n* 

What  do  you  think? 

I2-* 

m  Bia  ■  i  d  e- 

H 

There  is  a  9.1%  chance  "Between  8  0 

H 

-6.5"  will  be  the  answer 

*0 

X 

* 

Oa  IS  Dais  OelT 

Nicfccn  ratings  are  audience  nwsswemenl  systems 
tlerc-Iaped  by  Nifr]$en  Media  fttsrtaiilft.  pi  an  cIIhL  CO 

■d-Nerm.n?  e^e  aydeerce  Eire  and  composition  of 

lelnvcsian  prpfr attuning  in  (he  Umtrt 

Figure  8.  Screenshot  1  from  the  Prediction  Market  Simulations 

(inklingmarkets.com) 


Buying  Shares 

•  Buying  shares  means  you  think  this  answer  will  happen. 

•  The  more  shares  you  buy,  the  stronger  you  feel  this  answer  will  happen. 

•  If  the  answer  happens,  you  will  make  money.  If  the  answer  does  not  happen,  you  will  lose  money. 

answer  won't  happen  purchase  price  answer  will  happen 

so  A - ■  sioo 

losses  profits 

•  Your  cost  is  the  price  you  will  pay  for  buying  that  amount  of  shares. 

•  The  estimated  new  price  is  what  impact  you  will  have  on  the  price.  Remember,  by  buying  you  are  saying  the  probability  this 
will  occur  is  higher  than  the  current  price,  so  we  raise  the  price  according  to  how  strongly  you  (and  others)  feel. 

Figure  9.  Screenshot  2  from  the  Prediction  Market  Simulations 
(inklingmarkets.com) 


Marketplace  Stats 

Marketplace  performance  metrics,  this  week's  leaderboards  and  the  all-time  best  traders. 


T otal  number  of  traders 

17 

T otal  number  of  trades 
236 

Trades  this  week 

236 

Number  of  markets  running 

0 

Markets  completed 

0 


Most  frequently  traded  market  last  7  days 
no  data  yet 


Most  frequently  traded  market  all  time 

What  will  be  the  Neilson  Ratings  for  the  NBC  Sunday  Football  Game  (Oct  25)? 


Most  trades  this  week 

juhnltess  (bi) 


Most  trades  all  time 

johnhess  (8i) 


Biggest  price  increase  in  last  7  days 

Between  9.6-10.0  fiorn  What  will  be  the 
Neilson  Ratinqs  for  the  NBC  Sunday 
Football  Game  (Oct  25)7  ^  109.9% 


Biggest  price  decrease  in  last  7  days 

Between  1 3.0  -  16.0  from  What  will  be  the 
Neilson  Ratings  for  the  NBC  Sunday 
Football  Game  (Oct  25)9  ^  73.26% 


Figure  10.  Screenshot  3  from  the  Prediction  Market  Simulations 

(inklingmarkets.com) 
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Prediction  Markets:  History  &  Purpose 


Prediction  markets  are  speculative  markets  for  predicting  outcome  of 
uncertain  future  event.  E.g.:  Presidential  elections  or  Printer  sales. 

Buy  and  Sell  contracts  for  future  event 
Incentivizes  participants  to  gather  and  reveal  information 


Provide  dynamic  reflection  of  wisdom  of  group  as  a  whole 


Good  Answer 


Better  Answer! 


http://www.crowdclarity.com/leammore.htm 


Prediction  Markets:  Value  Proposition 


Probable  limitations  of  Possible  value  propositions 

existing  cost  methods  of  Prediction  Markets 


Technical  factors 

rather  than  “soft” 


ractors 


Information  leveraged  from 

^0^  diverse  sources 

< 

Incentivize  Traders 


Not  dynamically 
updated  as  the 

program  evolves 


Frequent  sampling  of 

information 


Few  decision  makers 

\ 

under  time  pressure  or 

.... 

biased 

:  ■  , 

1  :  ■ 

_ _ _ J 

Mitigation  of  biases 

Shift  of  focus  from  estimating 

by  individuals  to  groups 
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Prediction  Markets:  Examples 


Companies  Using  Collective  Intelligence  Software 

125 


Kerry  versus  Bush  Aggregate  Probabilities* 


2003  2004  2005  2006  2007  2008  2009 


Boeing  787  -  When  will  the  Boeing  787  make  its  maiden  flight? 
Boeing  787  to  make  maiden  flight  before  midnight  ET  on  31  Dec  2009 


Last  Price: 

You  can  buy 
this  at  47.0 

You  can  sell 
this  at  36.5 

Explain 


47.0 


|  Trade  Now 


Afghan  Politics  -  Who  will  become  President  of  Afghanistan  in  2009  Presidential  Election? 
Hamid  Karzai  to  be  sworn  in  as  President  of  Afghanistan  following  the  2009  election 


Last  Price: 

You  can  buy 
this  at  0.0 

You  can  sell 
this  at  85.0 

Explain 


80.0 


|  Trade  Now~ 


■  Closing  Price:  80 

"T; 

5 

Sep  09  Oct 
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Prediction  Markets:  Management  Tool 


Current  demand  for  better 
forecast  tools 


Information  through  Emails 


Organizing  &  Integrating  tacit 
information 


Capture  direct  and  indirect  factors  impacting  cost 


Low  demand  on  staff  time  and 
resources 


Easy  Implementation  through 
customized  tools 


Secure  IT  Environment 
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Prediction  Markets:  Structure  &  Types 


Marketplace 


Winner  Takes  All 


Index 


Spread 


Prediction  Markets:  General  Principles 


- \ 

Unbiased  tacit  knowledge  of  crowd 

- \ 

Well  -  incentivized  environments 

— 

Non  -  hierarchical  environment 

— 


Well  defined  end  dates  or  closing  criteria 


_ 

— 

Mutually  exclusive  choices  with  a  definitive  outcome 
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Prediction  Markets:  Case  Study  -  SWCS 


Design  Components  to  consider 

Stocks 

Marketplace 
T  raders 


SWCS  is  a  planned  replacement  to  the  MK  8 
MOD  1  SEAL  Delivery  Vehicle. 


IOC  FY  14 
FOC  FY  16 


Shallow  Water  Combat  Submersible 


Implement  prediction  markets  to  surface  potential  program  risks,  and  generate 
cost  /schedule  estimates  as  a  supplement  to  existing  estimation  methodologies 


Collaboration  with  MITRE 
to  share  best  practices: 
Hanscom  Air  Force  Base 


Prediction  Markets:  Benefits  to  SWCS 


- \ 

Benefit  from  the  prediction  markets  event  forecasts. 


Increased  involvement  of  the  participants  in  anticipating 

events. 


- \ 

Identifying  informal  information  channels  in  their 

organization. 


Agility 


Efficiency 


Transparency 


it 

”  7  1 
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Prediction  Markets:  SWCS  -  Design 


Stocks:  Designing  the  Questions 


Questions  of  the  First  Order:  Program  of  interest 

•  Will  SWCS  be  certified  by  August  1,  2010? 

•  The  cost  of  the  first  unit  will  be  $x. 


Questions  of  the  Second  Order:  Traders  and  the  _ 

trading  process 

•  What  is  your  motivation  for  trading?  (e.g.  to  win/  to  solve  the 
problem/  to  validate  my  knowledge) 

•  What  is  your  role  in  the  Organization? 


Questions  of  the  Third  Order:  Behavior  outside 
the  markets 


•  Did  trading  improve  your  confidence  in  your  opinion? 

•  Did  Prediction  Markets  encourage  you  to  change  your 
communication  habits? 
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Prediction  Markets:  SWCS  -  Design 


Marketplace 

—  Software  Environment 

•  Easy  to  access  and  use 

•  Commercially  available 

•  IT  security  considerations 

•  Dashboard  and  administrative  capabilities 

•  Share  information 

•  Close  or  open  new  markets 


—  Market  Manager 

•  Continuous  Double  Auction 

•  Automated  Market  Maker 


Prediction  Markets:  SWCS  -  Design 


Traders 


- \ 

Markets  will  be  implemented  for  duration  of  3  to  6  months 

k _ > 


- \ 

Traders  will  include  administrative  personnel,  sub¬ 
contracts,  engineers,  end  users,  consultants  etc 
s _ ; 


- \ 

Anonymity  of  traders  will  be  maintained 

k _ > 


- \ 

Participants  from  all  smaller  projects,  as  well  as  people 

from  other  departments 


MiUjJ  Jil  LUIIc  JiClltKjJ 

CA 


Acquisition  Research  Program:  Creating  Synergy  for  Informed  Change 


Prediction  Markets:  SWCS  -  Design 


Incentives 


- \ 

Declaring  winners  based  on  the  maximum  portfolio  value  at 

market  closure 


s 

■\ 


Maintaining  a  leadership  board  displaying  the  top  players 


Results  from  the  market  be  used  by  decision  makers  and  have 

an  influence  on  real  outcomes 


Recognition 


Opportunity  to  Contribute  Social  Competition 


a 
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Prediction  Markets:  Design  Principles 


Incentives  based  on  constant  participation  and  ability  to 

predict  accurate  results 


Ample  historical  data 


Mix  of  hard  and  fun  questions 


Anonymity  of  Participants 
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Prediction  Markets:  Design  Principles 


Ensure  liquidity  and  avoid  volatility 


j 


Support  of  senior  management 


Compatibility  with  political  and  regulatory  contexts 


Review  of  market  with  regular  reports 


Length  of  market  and  closure  of  questions 


IpUNfsJE 

'  88r~ 
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Prediction  Markets:  Q  &  A 


Acquisition  Research  Program;  Creating  Synergy  for  Informed  Change 


friUill  PuSi  ftrJll  LLllt  Hi'fchiPihl 

Mtmlcrey,  <l\ 


17 


Prediction  Markets:  Backup 
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Prediction  Markets:  SWCS  -  Design 


Sample  Risk  Mapping  for  First  Order  Questions 


Question  Type 

First  Order  Questions 

Measured  Risk 

Alternate  way  of  asking  the  question 

Type  1 

Will  SWCS  be  certified  by  August  1, 
2010? 

Schedule  Overrun 

When  will  SWCS  be  certified? 

Or 

By  how  many  days  will  the  certification 
date  be  exceeded? 

Type  I 

The  cost  of  the  first  unitwill  be  $x. 

Cost  Overrun 

What  do  you  think  is  the  most 
significant  cost  driver  for  the  first  Unit? 

Follow  up  question: 

Based  on  the  below  ranking  of  cost 
drivers  for  the  first  Unit,  what  do  you 
think  the  cost  of  first  Unitwill  be? 

Type  1 

The  final  cost  of  the  program  will 
exceed  the  baseline  estimate  by  x%. 

Cost  Overrun 

Type  1 

What  will  be  the  2011  Fiscal  Year 
appropriation  for  the  SWCS? 

Cost  and  Schedule 

Type  II 

The  SWCS  program  will  be 
adequately  resourced  to  meet  its 
milestones? 

Resource  Allocation  Errors 

By  how  many  days/months  will  the 
milestone  be  missed!? 

Or 

Which  resource  do  you  think  is  most 
necessary  to  meet  the  program's 
schedule  objective? 

Or 

Given  the  available  resources/  will  the 
project  be  able  to  meet  its  planned 
schedule? 
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Prediction  Markets:  Pilot  Studies 


Sunday  Night  Football  :  Pilot  Study  I 


>  Group  of  students  asked  to  participate  in  a  private  marketplace  for  predicting 

the  Nielsen  Rating  (NBC  -  USA)  for  Sunday  Night  Football  (Oct  25) 

>  Time  -  5  days,  Market  close  30  minutes  prior  to  game 

>  Winning  Criteria  -  Generate  maximum  worth 

>  Incentive  -  Points  for  Class  Grade  and  a  popular  book  for  winner 

>  Minimum  of  four  trades  per  person 

>  Each  participant  allotted  $5000  for  trading 


What  will  bethe  Nielsen  Ratings  for  the  NBC  Sunday 

Football  Game  (Oct  25]? 

■  a-  flit*  ■  ra+ 

Z2  I?*  ■  but  ■  i  □  6- 

What  do  you  think? 

M 

There  is  a  9.1%  chance  "Between  8  0 

SC 

-B. 5"  will  b&  the  artsw&r. 

Dan  OalS  Oat? 

Nielsen  ratings  ve  audience  messuferwrl  5-yslwns 
dew’aped  by  Nirijen  Media  ftisdaoHi.  Eh  an  ClfbA  ED 
determine  the  suilence  sire  anC  cnn^Krsitmn  of 
leltiMiitm  in  (ho  Umt*d 

Market  Information 


www .  inklingmarkets  .com 
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Prediction  Markets:  Pilot  Studies 


Sunday  Night  Football  :  Pilot  Study  I 


>  Each  rating  range  assigned  equal  probability  (9.09%)  to  start  with  (1 1 
ranges) 

>  Every  time  a  user  buys  a  share  for  a  particular  idea,  price  goes  up.  Similarly, 
each  time  users  sell  a  share,  price  goes  down.  Your  account  will  also  be 
credited  or  debited  based  on  your  choice. 

>  Direct  correlation  between  price  and  probability  -  share  price  of  $9.09 
indicates  9.09%  chance  of  that  particular  rating  range. 


What  will  be  the  Neilson  Ratings  for  the  NBC  Sunday  Football 
Game  (Oct  25)? 


Buying  Shares 


Select  a 

GcHwocn  S  fr-8  5 
Hfllw&en  SB-9  0 
^  1-9  5 

Satwaen  9  6- lfl  0 
Bf. tfrmh  tO  MO  5 
Between  1 0  0  ■  110 
11  M2.0 

Between  T2  Cl  •  13  0 
HctauGii  13  0  -  Its  0 
Cvor  16  0 
Betow  0  0 


S909 
S909 
59  W 
59  m 
59  09 
S9tfJ 
59  09 
$9  09 

99.09 
$9  09 


0.00 
000 
000 
000 
0  00 
0.00 
000 
0.00 
ooo 
000 
000 


•  Buying  shares  means  you  think  this  answer  will  happen. 

•  The  more  shares  you  buy,  the  stronger  you  feel  this  answer  will  happen. 

•  If  the  answer  happens,  you  will  make  money.  If  the  answer  does  not  happen,  you  will  lose  money. 


answer  won’t  happen  purchase  price 

soB"1  ■  - . A - 


answer  will  happen 

□  si  00 


profits 


•  Your  cost  is  the  price  you  will  pay  for  buying  that  amount  of  shares. 

•  The  estimated  new  price  is  what  impact  you  will  have  on  the  price.  Remember,  by  buying  you  are  saying  the  probability  this 
will  occur  is  higher  than  the  current  price,  so  we  raise  the  price  according  to  how  strongly  you  (and  others)  feel. 


www .  inklingmarkets  .com 
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Prediction  Markets:  Pilot  Study  I  Result 


>  Around  236  trades  conducted  in  total  by  17  traders 

>  Highest  Rating  Range  (9.6  -  10)  (19%  chance),  followed  by  range  (9.1  -  9.5) 
(17.3%  chance);  Actual  rating  -  9.1 

>  World  Series  game  ran  into  overtime,  reducing  the  ratings  for  Sunday  Night  Football 

Whatwill  be  the  Nielsen  Ratings  for  the  NBC  Sunday  Football 
Game  (Oct25)? 
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Prediction  Markets:  Pilot  Studies 


Monday  Night  Football  :  Pilot  Study  II 

>lnstead  of  ranges,  participants  now  saw  only  one  rating.  They  had  to  predict  if 
the  rating  as  per  their  opinion  was  to  be  higher  or  lower.  Based  on  their 
responses,  the  price  or  rating  was  adjusted  automatically. 

>The  initial  rating  set  was  10 

>The  price  and  rating  scale  was  1:1,  i.e.  a  price  of  $10.00  meant  the  current 
projected  value  of  rating  was  10.00 


What  will  be  the  Nielsen  Rating  (Cable  TV  -  US)  for 
Monday  Night  Football  on  ESPN  (Nov 2)? 


Prediction  Markets:  Pilot  Study  II  Result 


Monday  Night  Football  :  Pilot  Study  II 

•Around  116  trades  were  conducted  by  13  participants.  The  final  rating 
predicted  by  the  market  was  10.00 

•There  was  a  huge  variation  in  the  ratings  predicted,  with  the  highest  value 
being  33.1  and  the  lowest  value  being  0.6 

•The  actual  rating  came  out  to  be  7.7  which  was  not  close  to  the  market  value. 


Prediction  Markets:  Observations 


Behavioral  Observations 

increased  participation  due  to  Internal  competition 
^Attempts  at  market  gaming 
>Trying  to  make  use  of  First  mover  advantages 
> Forming  alliances  to  manipulate  the  market 

>High  incentive  to  participate  led  participants  to  gain  more  knowledge 
on  the  subject. 

>Use  of  statistical  models  by  some  participants  to  forecast  ratings 
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Prediction  Markets:  Observations 


Useful  insights  related  to  implementation 

>Ensure  enough  liquidity 

>  Avoid  volatility  by  choosing  the  right  scale  for  trading  in  market 
>Ease  of  Use 

>Ensure  regular  information  update 
>Leads  to  increased  awareness  and  interest 

>  Align  incentives  with  the  participants’  interests 


Acquisition  Research  Program;  Creating  Synergy  for  Informed  Change 


llUSiftr  jJ  LUlIc  Hi'thlPlhl 
Mtmlcrey,  <l\ 


26 


Prediction  Markets:  Case  Study  -  SWCS 


Stocks:  Stocks  traded  or  the  questions  answered  give  an  indication  of 
the  beliefs  of  the  participants,  and  the  number  of  stocks  traded  show  the 
level  of  confidence  in  their  beliefs. 


>  Type  I  questions:  Asking  these  questions  can  directly  provide  the 
required  information. 

Qh  ->A 

>  Type  II  questions:  Asking  these  questions  can  indirectly  provide 
the  required  information,  and  a  series  of  questions  might  be  required 
for  getting  the  information  directly. 

Q2^Q’ 
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